Skip to content

Migrate X-Chain from LevelDB to Firewood and PebbleDB#3732

Open
joshua-kim wants to merge 1 commit intox-chain-merkle-db-firewoodfrom
x-chain-merkle-db
Open

Migrate X-Chain from LevelDB to Firewood and PebbleDB#3732
joshua-kim wants to merge 1 commit intox-chain-merkle-db-firewoodfrom
x-chain-merkle-db

Conversation

@joshua-kim
Copy link
Copy Markdown
Contributor

@joshua-kim joshua-kim commented Feb 20, 2025

Why this should be merged

Implements a database migration from the legacy database format into a database format that supports a merkle-ized prefix of state

How this works

Changes db schema

How this was tested

Added a migration unit test + tested state migration on Fuji

[03-20|16:01:06.597] INFO <X Chain> avm/vm.go:173 VM config initialized {"config": {"network":{"max-validator-set-staleness":60000000000,"target-gossip-size":20480,"push-gossip-percent-stake":0.9,"push-gossip-num-validators":100,"push-gossip-num-peers":0,"push-regossip-num-validators":10,"push-regossip-num-peers":0,"push-gossip-discarded-cache-size":16384,"push-gossip-max-regossip-frequency":30000000000,"push-gossip-frequency":500000000,"pull-gossip-poll-size":1,"pull-gossip-frequency":1500000000,"pull-gossip-throttling-period":10000000000,"pull-gossip-throttling-limit":2,"expected-bloom-filter-elements":8192,"expected-bloom-filter-false-positive-probability":0.01,"max-bloom-filter-false-positive-probability":0.05},"index-transactions":false,"index-allow-incomplete":false,"checksums-enabled":false}}
[03-20|16:01:06.597] INFO <X Chain> avm/vm.go:598 fee asset is established {"alias": "AVAX", "assetID": "U8iRqJoiJm8xZHAacmvYyZVwqQx6uDNtQeP3CQ6fcgQk3JqnK"}
[03-20|16:01:06.597] INFO <X Chain> avm/vm.go:257 address transaction indexing is disabled
[03-20|16:01:06.599] INFO <X Chain> snowman/engine.go:96 initializing consensus engine
[03-20|16:01:06.599] INFO <X Chain> bootstrap/bootstrapper.go:320 starting bootstrap
[03-20|16:01:06.601] INFO <X Chain> bootstrap/bootstrapper.go:600 executing transactions
[03-20|16:01:06.601] INFO <X Chain> queue/jobs.go:223 executed operations {"numExecuted": 0}
[03-20|16:01:06.601] INFO <X Chain> bootstrap/bootstrapper.go:612 executing vertices
[03-20|16:01:06.601] INFO <X Chain> queue/jobs.go:223 executed operations {"numExecuted": 0}
[03-20|16:01:06.603] INFO <X Chain> avm/state_migration.go:143 starting state migration
[03-20|16:04:53.784] INFO <X Chain> avm/state_migration.go:321 migration complete
[03-20|16:04:53.790] INFO <X Chain> proposervm/vm.go:209 initialized proposervm {"state": "after fork", "forkHeight": 1, "lastAcceptedHeight": 32356}
[03-20|16:04:53.790] INFO <X Chain> bootstrap/bootstrapper.go:182 starting bootstrapper {"lastAcceptedID": "8o6VfSzPHhjLXwr7FEZDxuVAwXYVCx6dP7vrwLoW9mwYRPZi4", "lastAcceptedHeight": 32356}
[03-20|16:05:42.318] INFO <X Chain> bootstrap/bootstrapper.go:387 starting to fetch blocks {"numKnownBlocks": 99, "numAcceptedBlocks": 1, "numMissingBlocks": 100}
[03-20|16:05:42.857] INFO <X Chain> bootstrap/storage.go:194 executing blocks {"numToExecute": 45}
[03-20|16:05:43.116] INFO <X Chain> bootstrap/storage.go:186 executed blocks {"numExecuted": 45, "numToExecute": 45, "halted": false, "duration": "258.784792ms"}

Need to be documented in RELEASES.md?

Yes

@joshua-kim joshua-kim changed the title X chain merkle db Merkle-ize X-Chain State Feb 20, 2025
Comment thread vms/components/avax/utxo_state.go Outdated
@joshua-kim joshua-kim changed the base branch from master to merkle-db-config March 4, 2025 22:37
@joshua-kim joshua-kim force-pushed the x-chain-merkle-db branch 3 times, most recently from e89db38 to 20ae963 Compare March 6, 2025 16:27
@joshua-kim joshua-kim force-pushed the x-chain-merkle-db branch 7 times, most recently from db6f8b7 to 0505956 Compare March 11, 2025 18:52
Comment thread database/memdb/db.go
Comment thread vms/avm/state/state.go
Comment thread vms/avm/state/state.go Outdated
@joshua-kim joshua-kim self-assigned this Mar 18, 2025
@joshua-kim joshua-kim added the vm This involves virtual machines label Mar 18, 2025
@github-project-automation github-project-automation Bot moved this to Backlog 🗄️ in avalanchego Mar 18, 2025
@joshua-kim joshua-kim moved this from Backlog 🗄️ to In Progress 🏗 in avalanchego Mar 18, 2025
@joshua-kim joshua-kim marked this pull request as ready for review March 18, 2025 20:31
@github-actions github-actions Bot added the lifecycle/stale Inactive for 60 days label Apr 27, 2025
@joshua-kim joshua-kim moved this to Ready 🚦 in avalanchego May 1, 2025
@StephenButtolph StephenButtolph removed the lifecycle/stale Inactive for 60 days label May 1, 2025
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 1, 2025

This PR has become stale because it has been open for 30 days with no activity. Adding the lifecycle/frozen label will cause this PR to ignore lifecycle events.

@github-actions github-actions Bot added the lifecycle/stale Inactive for 60 days label Jun 1, 2025
@joshua-kim joshua-kim added lifecycle/frozen and removed lifecycle/stale Inactive for 60 days labels Jun 2, 2025
@joshua-kim joshua-kim requested a review from rrazvan1 as a code owner July 22, 2025 16:12
Comment thread vms/avm/state/state.go Outdated
@joshua-kim joshua-kim force-pushed the x-chain-merkle-db branch 2 times, most recently from deeefd1 to 81b3617 Compare September 25, 2025 20:06
@joshua-kim joshua-kim moved this from Ready 🚦 to In Progress 🏗️ in avalanchego Oct 27, 2025
@joshua-kim joshua-kim force-pushed the x-chain-merkle-db branch 8 times, most recently from ef70c9a to 072f33e Compare October 30, 2025 19:16
@blacksmith-sh

This comment has been minimized.

@joshua-kim
Copy link
Copy Markdown
Contributor Author

Need to come up with a way to break up this diff. Currently I think that we could break this into:

  1. Introducing read-only shared memory
  2. firewood package
  3. Diff in state

@blacksmith-sh

This comment has been minimized.

Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lifecycle/frozen vm This involves virtual machines

Projects

Status: In Progress 🏗️

Development

Successfully merging this pull request may close these issues.

Merkle-ize X-Chain

4 participants